Method of content aware image resizing

ABSTRACT

Audiovisual content presentation to users has evolved from users receiving hardcopy printed materials to their searching and retrieving information by accessing any of hundreds of millions of web sites and billions of web pages. User retrieval being performed on a wide variety of platforms from high performance PCs to low performance cellular telephones. Accordingly substantial limitations exist in initially displaying this audiovisual content as well as when users dynamically manipulate browser dimensions or move through the content. Further additional limitations exist for those authoring both online and traditional content to manipulate sourced content to provide the published content. As such a requirement exists for dynamically resizing images that respects the information content within the audiovisual content. Embodiments of the invention provide for content aware resizing of audiovisual content both within authoring and user environments, and in dependence upon device characteristics such as display, processor, power, etc.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of co-pending U.S. patent applicationSer. No. 12/691,765 filed Jan. 22, 2010. The aforementioned relatedpatent application is herein incorporated by reference in its entirety.

FIELD OF THE INVENTION

This invention relates to image processing and more specifically tocontent aware image re-dimensioning.

BACKGROUND OF THE INVENTION

In the last fifteen years or so accessing, generating and exchanginginformation has fundamentally shifted for Governments, commercialenterprises, private and public organizations, and the general public.In those fifteen or so years the Internet has gone from a nicheapplication to an essential element of the lives of most individuals inthe developed world. As of Jul. 1, 2009 it was estimated that the numberof Internet users had exceeded 1.67 billion people out of a worldpopulation of approximately 6.8 billion, i.e. 25% of the world'spopulation. These users are accessing information contained inapproximately 22 billion pages hosted on over 110 million websites(http://www.domaintools.com/internet-statistics).

Over the same period of time how the Internet is accessed has shifteddramatically as well. No longer are users sitting at desktop personalcomputers (PCs) in front of 15″ or 17″ CRT displays interfaced to largemetal cases hosting for example a single Intel® 486 processor operating50 MHz or 100 MHz with 32 MB memory with a 16 GB hard-drive accessingdial-up connectivity at 56 kb/s. Today their desktop PC is most likelyto be a laptop PC working alone or in conjunction with a LCD display ofdimension 17″, 19″, 21″, etc up to 32″ or more for graphical designersallowing them to unplug and move to another location to continue workingThis laptop for example containing an AMD Athlon™ Dual-Core 2.00 GHzprocessor with 4 GB memory, a 500 GB hard-drive, and with Internetconnectivity at 5 Mb/s, 10 Mb/s or more through wireless WiFi (IEEE802.11) or WiMAX (IEEE 802.16) interfaces.

Additionally a multitude of other portable electronic devices nowprovide their users with Internet access including for example personaldigital assistants (PDAs) and cellular telephones (e.g. Apple iPhone,Research in Motion's Blackberry, Palm Pre, Samsung Chocolate), gamingconsoles (e.g. Microsoft Xbox, Nintendo DSi, Nintendo Wii), andaudiovisual media players (e.g. Apple iPod). Accordingly users canaccess the Internet essentially anywhere and anytime with one of severaldevices they typically posses. Further recent device developments suchas the Apple iPhone® with integrated silicon MEMS devices allow fordynamic rotation of the mobile device display between landscape andportrait formats as the user rotates their device. Further, operatingsystems such as Microsoft's Windows and Apple's MAC OS X allow users todynamically change the size and effective orientation of web pages ontheir computers, and newer introductions such as Microsoft Window 7allow users to dynamically move and display directly content from theirlaptop PC to another device such as another laptop, television, PDA etc.

As a result the original consideration of images on mobile devices assimply wallpaper and screen savers or web site content as beingdisplayed on large portrait orientated PC displays has been destroyed.Audiovisual content posted to the Internet within any web page isdynamically accessed, dynamically adjusted, and is highly manipulated. Anews image may be accessed within seconds by millions of users withdisplays from typical cellular telephone 240×320 and 320×480 pixeldisplays of 2.S″ or 3.2″ through to IS″ or 17″ displays of 1920×1080psupporting HDTV and above to 32″, 42″ LCD, plasma displays, andprojectors as users employ their televisions as monitors.

Further users expectations have increased during this time. Applicationssuch as Microsoft Word and Corel WordPerfect have evolved from beingsimple word processing applications to entry level desktop publishingsuites supporting graphics and audiovisual content and the generation ofweb pages. At the same time desktop publishing software has expanded tofacilitate direct handling of XML, HTML languages, multiple interfacesto digital audio, digital photo, and digital video applications andallowing direct publication in printed formats, secured digital content,and web content.

However, despite all these advances the content published onto Internetweb pages is in the vast majority of cases fixed, even from leadingcontent providers such as Yahoo and Google. Hence, as the viewing useradjusts the dimensions of their web page, for example allowing them toview the Internet content whilst working on another application withouthaving to move from one application to another, then essentially theirweb page acts similar to a window adjusting the amount of the web pagethey can view but the audiovisual content is typically fixed in size. Inthe other cases, for example Google image search, the content isadjusted to a limited extent according to the dimensions of the webbrowser page, for example the number of images across the web browserpage changes. However, the image sizes remain constant and the user mustnow scroll further to view all the images and move to the next page. Inothers the page layout adjusts to display the text according to the webbrowser page size but again the dimensions of the image have been fixed.Today image manipulation in respect of adjusting displayed dimensions ofan image is essentially limited to the desktop publisher's domain whengenerating the web page content. The user's ability to control thedisplay of the web page content is limited to either adjusting the webbrowser page size or adjusting the zoom that the web browser displayscontent with.

It would be beneficial for audiovisual content presented to a user to bedynamically displayed according to a variety of factors including butnot limited to the dimensions of the web browser page, image displaydevice dimensions, and image display device resolution for example. Inthis manner disadvantages of the prior art that will become evident inthe descriptions of these approaches will be removed.

Amongst the earliest prior art techniques for image adjustment toreflect a change in displayed dimensions is cropping, such as shown inFIG. 1, where two desktop publisher snapshot images 100 and 150 areshown. First desktop publisher snapshot image 100, from Adobe PhotoshopLightroom® shows an image of a bride 110 together with a croppedhighlighted region 120 which the user will select as the cropped imageto employ. Similarly second desktop publisher snapshot image 150, fromAdobe Photoshop shows a cityscape 160 together with a cropped cityscaperegion 170 which the user has selected as the cropped image to employ.Second desktop publisher snapshot image 150 also has icon 180 thatprojects an automatically generated mask onto the cityscape 160 ateither a predetermined pixel count or physical dimension. However, thisprior art approach only works to reduce an image dimension, it cannotscale the image up, and if automatically generated may removesignificant content in the image. Cropping does not scale the sourceimage even when reducing the displayed dimensions and has typically beenlimited to date therefore to desktop publishing.

Within the prior art there are many approaches to automate the croppingoperation by detecting content and cropping in dependence of thecontent. Examples include A. T. Schowkta in U.S. Pat. No. 7,133,050entitled “Automated Image Resizing and Cropping”, Suh et al in“Automated Thumbnail Cropping and its Effectiveness (UIST'03 Proc. 16thACM Symposium User Interface Software and Technology, ACM Press, NewYork, pp. 95-104, 2003), A. Santella et al in “Gaze-Based Interactionfor Semiautomatic Photo Cropping” (Proc. SIGCHI Conference on HumanFactors in Computing Systems, pp. 771-780, 2006) and E. G. Callway in USPatent 2007/0,152,990 entitled “Image Analyzer and Adaptive ImageScaling Circuit and Methods”.

Within the prior art such cropping methodologies have been employed inconjunction with linear and non-linear scaling methodologies to provideimages of variable size. Linear and non-linear scaling allows thegeneration of images that are both larger and smaller than the originalwhilst cropping adjusts the image content. Such a non-linear techniquebeing shown in FIG. 2 by resizing tool window 200, as provided by SBSoftware (Nonlinear Image Resizing Tool, Version 1.0,www.sb-software.com). As shown within resizing tool 200 an originalimage 210 of dimensions 747×923 pixels has been selected for resizing toresized image 220 of dimensions 1024×768 pixels representing an aspectratio change from 0.81:1 to 1.33:1. As indicated by resizing settingtoolbar 230 the user can apply nonlinear factors that range fromsqueezing the centre of the image and stretching the edges of the imagethrough to the reverse of stretching the centre of the image andsqueezing the edges of the image. Such a non-linear scaling whilst animprovement over linear scaling in many instances can still result inunnatural images, particularly as the human visual process is highlysensitive to distortion and non-linearity.

Extensions of this technique to reduce such visual irregularities andreduce the user perceptions that image manipulation has been undertakenhave included A. Soroushi in U.S. Pat. No. 7,355,610 entitled “Methodand Apparatus for Expanding Image Data to Create Enlarged Images forDisplay”, Y-H. Lee in US Patent Application 2007/0,147,708 entitled“Adaptive Image Size Conversion Apparatus and Method Thereof’, and C-H.Chou in US Patent Application 2007/0,104,394 entitled “Method and Systemfor Digital Image Magnification and Reduction.” However, such whilstaddressing the automation aspect of dynamically adjusting images todifferent display devices or varying web browser page dimensions theyhave drawbacks in terms of requiring significant processing complexityeven if they can be implemented in the firmware of devices or requireadditional specific hardware.

It would be apparent that a requirement for a solution addressing highvolume consumer applications of image display would be beneficiallyprovided without requiring additional hardware and in a software Ifirmware form that operates within a wide range of portable devices withvarying processing capabilities. Further such firmware shouldbeneficially operate rapidly to provide real time image resizing andwith low power consumption to extend the portable device lifetime to theuser. Such a focus within the prior art is typically absent as mostprior art applications have focused to desktop publishing typeapplications such as Adobe Photo shop, Corel PhotoShop, MicrosoftPowerPoint, and Microsoft Publisher for example wherein the user isprimarily authoring and generating content for publication either inphysical or online media formats. Referring to FIG. 3 there is presentedan image scaling flow according to the prior art of S-H Lee in US PatentApplication 2008/0,019,439 entitled “Apparatus and Method for LowDistortion Display in a Portable Communication Terminal”. As shown infirst step 300A an image 310 has been received by a portable device, notshown for clarity, for display that requires resizing. Accordingly theprocess of Lee divides the image 310 in second step 300B to a pluralityof image segments 321 through 327 in preparation for applying thetransformation to each image segment 321 through 327. In third step 300Ca linear or non-linear scaling is applied to each image segment 321through 327 thereby generating scaled image segments 331 through 337.The scaling applied to each of image segment 321 through 327 to generatescaled image segments 331 through 337 being different such that thecontent is scaled to an increased percentage of the image to bedisplayed to the user but is done so in a manner that is supposed toreduce perceived distortion.

However, Lee applies a predetermined scaling according to a mathematicalfunction, for example a cosine function, such that weighting in thescaled image is given to the central portion of the content which isexpanded and the outer portions are reduced when the overall image is tobe reduced dimensionally. Whilst other mathematical functions may beemployed such as a sine, hyperbolic tangent, sinc etc for example theappropriate mathematical function should be determined by the content ofthe image which requires in an automatic scaling application, that theimage be first processed to determine the distribution of content andhence appropriate function to apply. Equally, Lee only teaches applyingthe function in one dimension whereas it would be beneficial to providethe methodology in two dimensions when considering the target portabledevices etc. Other examples within the prior art include P. O. Vale inU.S. Pat. No. 7,385,615 entitled “System and Method for Scaling Imagesto Fit a Screen on a Mobile Device According to a Non-Linear ScaleFactor”.

A further alternative is taught by H. Chao et al in US PatentApplication 2008/0,095,470 entitled “Digital Image Auto-Resizing” andshown schematically in FIG. 4 as applied to an initial image 410. Asshown Chao teaches that the image is broken into two portions, a firstportion 420 where the content will be scaled at a first scaling factor,and a second portion 440 which will be scaled at a second scalingfactor.

Accordingly first portion 420 is broken into four elements, first tofourth elements 421 through 424 respectively, which will be scaled tofit the new overall window to present the scaled image 460 but isperformed in a manner to reduce the reduction in the portion of thescaled image given to the second portion 440. Hence, first element 421and fourth element 424 would be scaled only in the horizontal axiswhilst second element 422 and third element 423 would be scaled only inthe vertical axis. As such the scaled replicas of first to fourthelements 421 through 424 respectively are combined to form scaled firstportion 430. The second portion 440 is scaled to generate scaled secondportion 450 and is then combined with scaled first portion 440 togenerate the scaled image 460 to be presented to the user. Again adrawback of Chao is that selecting the portions of the image, namelyfirst and second portions 420 and 440 respectively, can significantlyimpact the resultant scaled image 460 and the viewer's perception orsatisfaction as a result. Other examples of such blocked scaling ofimages include K. Berkner et al in U.S. Pat. No. 7,548,654 entitled“Header Based Scaling and Cropping of Images Compressed UsingMulti-Scale Transforms” and S J. Kaasila et al in U.S. Pat. No.7,287,220 entitled “Methods and Systems for Displaying Media in a ScaledManner and/or Orientation”.

Extensions of such cutting, scaling and re-pasting include thosereported by V. Setlur et al in “Automatic Image Re-Targeting” (Proc.18th ACM Symposium on User Interface Software and Technology, pp.153-162, 2005), J. Jia et al in “Drag-and-Drop Pasting” (Proc. SIGRAPH2006, Vol. 25, No. 3, pp. 631-637July 2006), J. Wang et al in“Simultaneous Matting and Compositing” (Microsoft Technical ReportMSR-TR-2006-63, May 2006), C. Jacobs et al in “Adaptive Grid-BasedDocument Layout” (Proc. ACM SIGGRAPH, pp. 838-847, 2003), W. T. Freemanet al in U.S. Pat. No. 6,919,903 entitled “Texture Synthesis andTransfer for Pixel Images”, and I. Clarke et al in US Patent Application2006/0,072,853 entitled “Method and Apparatus for Resizing Images.”

A further extension of this approach within the prior art was describedby B. S. Hallberg et al in U.S. Pat. No. 6,563,964 entitled “ImageDown-Sampling Using Redundant Pixel Removal” wherein the image to bereduced in size was non-uniformly down-sampled to remove aliasing withinthe high spatial frequency information content such that low spatialfrequency information content is preferentially removed. This requiredthat the image be processed by a spatial frequency estimator thatcompared groups of pixels in order to produce a classification of theimage. Subsequently a path generator and path scorer analyze potentialdeletion paths within the image and the path with highest score, the onegiving minimal distortion and aliasing, is selected for pixel removal.This process being repeated until a desired number of image rows and/orcolumns have been removed. As such Hallberg teaches that the entireimage is arbitrarily analyzed rather than the preceding prior artwherein sampling of the image for determination of scaling waspredetermined by applying a mask, template or mathematical function.However, Hallberg as noted only addresses reduction and is primarilyfocused to the problem of reducing the display of textual basedinformation such as directory listings etc in applications such asWindows Explorer as the display type varied rather than arbitrary windowgeneration as users adjust web browser pages etc.

The approach of Hallberg was extended by S. Aviden et al as reported inU.S. Pat. No. 7,477,800 entitled “Method for Re-Targeting Images” andtheir publication “Seam Carving for Content Aware Image Resizing” (ACMTransactions on Graphics SIGGRAPH 2007, Volume 26, Number 3, Article 10,July 2007). Aviden coined the term “seam carving” to refer to a simpleimage operator that provides adjustment of an image's size bycarving-out or inserting pixels in different parts of the image. Thedetermination of “seams” to carve or insert being made in respect of anenergy function that defines the importance of pixels. A “seam” beingdefined by a connected path of low energy pixels crossing the image fromone side to another representing the minimum energy path across theimage. Removal of these “seams” providing for reduction in the imagedimension in horizontal and/or vertical dimensions whilst insertion ofthese “seams” providing for expansion of the image. Aviden states thatthe image operator produces, in effect, a content-aware resizing of theimage.

Additional extensions of this work have been reported by M Klingemann(see flash blog http://www.quasimondo.com/archives/000652.php ofSeptember 2007) using an energy function generated through convolvingthe image with a blurred offset version of itself, the offset being afew pixels. H. Welles has also published open source implementations ofthe “seam carving” method of Aviden (see Ariadne and Seamstressalgorithms at http://seam-carver.sourceforge.net).

Aviden teaches that the digital image to be dimensionally adjusted isinitially converted into a so-called “energy map” wherein every pixel inthe image is mapped to a pixel within the “energy map,” Subsequently thecumulative energy for a continuous I-pixel wide “seam” is calculatedfrom one side of the image to the other side. The two preferred energyfunctions taught are outlined below in Equations 1 and 2. Aviden teachesthat no single energy function works well across all images but thatmost have similar ranges of resizing before visual artifacts areintroduced.

$\begin{matrix}{{e_{1}\left( {I\left( {x,y} \right)} \right)} = {{{\frac{\delta}{\delta \; x}{I\left( {x,y} \right)}}} + {{\frac{\delta}{\delta \; y}{I\left( {x,y} \right)}}}}} & (1) \\{{e_{HoG}\left( {I\left( {x,y} \right)} \right)} = \frac{{{\frac{\delta}{\delta \; x}{I\left( {x,y} \right)}}} + {{\frac{\delta}{\delta \; y}I\left( {x,y} \right)}}}{\max \left( {{HoG}\left( {I\left( {x,y} \right)} \right)} \right)}} & (2)\end{matrix}$

where I(x, y) is a particular pixel, and HoG (I(x, y)) is taken to be ahistogram of orientated gradients at every pixel (see N. Dalal et al“Histograms of Orientated Gradients for Human Detection” Intl. Conf.Computer Vision and Pattern Recognition, Vol. 2, pp 886-893). Avidenteaches using an 8-bin histogram computed over an 11×11 window around apixel for HoG(I(x, y)).

Referring to FIG. 5 the method of A divan is presented using imagestaken from the publication “Seam Carving for Content Aware ImageResizing” (ACM Transactions on Graphics SIGGRAPH 2007, Volume 26, Number3, Article 10, July 2007). A source image 510 is shown, and theintention is to change the aspect ratio from say 4:3 to 16:9. Applying aconventional linear scaling to source image 510 results in linear image520. Applying the method of “seam carving” of Aviden begins with thegeneration of the “energy map” 530 from source image 510. From thissingle “energy map” a horizontal seam map 540 is generated together withvertical seam map 550 that define the cost of removing a seam in eachdirection. Based upon the determination to remove either a horizontaland/or vertical seam a carved image 560 is generated. If the carvedimage 560 is not at the target image size then the process cycles backto recalculate the energy map 530 and repeats until the final imagedimension is achieved.

Aviden teaches that resizing an image from 240×320 pixels to 128×160pixels, such as reflects an image shifted from the inner display of aBlackberry Pearl Flip cellular telephone to it's outer display, would beachieved by removing 112 vertical ‘seams’ and 160 horizontal “seams”.Removal of each seam requires that the “energy map” is recalculated todetermine which “seam” is to be removed next. Accordingly the removal ofthe 112 vertical and 160 horizontal “seams” requires the generation of272 “energy maps” which is computationally intense, particularly so ifEquation (2) was employed. As such Aviden teaches that a designer mayauthor a multi-size image once and a client application depending uponthe image size needed performs the requisite number of “seam” removalsor additions such that the resizing can occur quickly in real time tofit the layout or display. The authoring being the computationallyintense generation of the large number of “energy maps” and processingof the “seam” determinations to generate the multiple image sizes. Theinformation relating to the multiple image sizes would for example bestored as a header within the image file. Such an approach of headerencoding being taught, albeit not in relation “seam carving” for exampleby K. Berkner in U.S. Pat. No. 7,548,654 as outlined supra.

However, a user accessing the Internet and retrieving images is notgoing to only access images generated by publishers with desktoppublishing software that included the “seam carving” information formultiple images embedded within. Further such an approach also affectseven the retrieval of audiovisual content by increasing the file size.As of mid-2009 the indexable web contained at least 22 billion pages(http://www.worldwidewebsize.com) hosted on over 110 million websites(http://www.domaintools.com/internet-statistics). Simply searching usingGoogle for images with “photo” returns over 700 million results whilstpopular social networking websites such as Facebook are reported at peaktimes to have 300,000 images uploaded a second by registered members. Itwould be evident that even if “seam carving” was introduced into allimage generating devices, such as desktop publishing software, digitalcameras, cellular telephones etc, by virtue of being embedded as part ofan international standard such as Portable Network Graphics (PNG),Tagged Image File Format (TIFF), and Motion Pictures Expert Group (MPEG)for example, that it would take a significant period of time to becomethe dominant format for digital audiovisual content accessible toInternet users.

Accordingly it would be beneficial to provide a method of resizingdigital images that was independent of their method of generation, i.e.portable consumer electronics or desktop software, independent of theplatform upon which the images were to be displayed, i.e. low costconsumer portable devices or laptop computers, the display they are tobe displayed upon, i.e. 128×160 pixel 1.8″ cellular telephone display,1600×900 pixel 17.3″ laptop, user activity such as flipping the AppleiPhone from a 320×480 pixel portrait orientation to 480×320 pixellandscape orientation in a fraction of a second, and the source imageformat.

It would be further beneficial if the method of resizing was alsocontent aware, i.e. provided scaling that did not remove significantimage elements or distort images at typical resizing factors unlessexpressly permitted by the user. Such permission being provided withindesktop publishing or image manipulation software such as Abode Photoshop, Corel Paint Shop Pro, Ulead Photo Impact for example. It would befurther beneficial if the method permitted the protection of contentduring resizing or explicitly weighted content for removal duringresizing or editing, was fast, and easily incorporated into the firmwareof devices as well as desktop publishing software.

SUMMARY OF THE INVENTION

It is an object of the present invention to obviate or mitigate at leastone disadvantage of the prior art.

In accordance with an embodiment of the invention there is provided acomputer implementable method comprising the steps of generating atleast one saliency map of a plurality of saliency maps, the at least onesaliency map characterized by at least a number of pixels equal to anumber of pixels within a source audiovisual content and each pixelwithin the at least one saliency map having a value associated with itwhich is determined in dependence upon determining at least a saliencyfor the corresponding pixel within the source audiovisual content, andgenerating at least one reduced saliency map of a plurality of reducedsaliency maps, each reduced saliency map associated with a saliency mapand generated by applying at least one predetermined scaling factor tothe at least one saliency map. The method further comprising the stepsof applying a path determination process to the at least one reducedsaliency map, the path determination process for identifying a sequenceof pixels within the at least one reduced saliency map that meet apredetermined condition, and modifying the source audiovisual content independence upon at least the sequence of pixels.

In accordance with another embodiment of the invention there is provideda computer implementable method comprising the steps of generating asaliency map characterized by at least a number of pixels equal to anumber of pixels within a source audiovisual content and each pixelwithin the saliency map having at least two values associated with it,one value determined in dependence upon at least a saliency for thecorresponding pixel within the source audiovisual content along a firstaxis of the image and the other value determined in dependence upon atleast a saliency for the corresponding pixel within the sourceaudiovisual content along a second axis of the image and generating areduced saliency map by applying at least one predetermined scalingfactor to the saliency map, each pixel with the reduced saliency maphaving at least first data generated in dependence upon at least the onevalue of a pixel within the saliency map associated with the pixel inthe saliency reduced map and second data generated in dependence upon atleast the other value of a pixel within the saliency map associated withthe pixel in the saliency reduced map. The method further comprising thesteps of applying a path determination process to at least one of thefirst data and the second data within the reduced saliency map, the pathdetermination process for identifying a sequence of pixels within thereduced saliency map that meet a predetermined condition, and modifyingthe source audiovisual content in dependence upon at least the sequenceof pixels.

In accordance with another embodiment of the invention there is provideda device comprising:

-   (a) an interface for receiving audiovisual content for presentation    to a user upon a display forming a predetermined portion of the    device, the audiovisual content characterized by at least a source    dimension being at least one of a physical dimension and a number of    pixels; and-   (b) a circuit including at least a processor and a memory for    executing a series of processes, the processes including at least:    -   (i) a display process for determining a target dimension for the        audiovisual content for presentation to the user; and    -   (ii) an image process for generating a modified image in        dependence upon at least the audiovisual content, the target        dimension, and the at least a source dimension, the image        process comprising the steps of:    -   (1) generating a saliency map characterized by at least a number        of pixels equal to a number of pixels within the source        audiovisual content and each pixel within the saliency map        having at least two values associated with it, one value        determined in dependence upon at least a saliency for the        corresponding pixel within the source audiovisual content along        a first axis of the image and the other value determined in        dependence upon at least a saliency for the corresponding pixel        within the source audiovisual content along a second axis of the        image;    -   (2) generating a reduced saliency map by applying at least one        predetermined scaling factor to the saliency map, each pixel        with the reduced saliency map having at least first data        generated in dependence upon at least the one value of a pixel        within the saliency map associated with the pixel in the        saliency reduced map and second data generated in dependence        upon at least the other value of a pixel within the saliency map        associated with the pixel in the saliency reduced map;    -   (3) applying a path determination process to at least one of the        first data and the second data within the reduced saliency map,        the path determination process for identifying a sequence of        pixels within the reduced saliency map that meet a predetermined        condition; and    -   (4) modifying the audiovisual content in dependence upon at        least the sequence of pixels to generate display audiovisual        content.

Other aspects and features of the present invention will become apparentto those ordinarily skilled in the art upon review of the followingdescription of specific embodiments of the invention in conjunction withthe accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described, by way ofexample only, with reference to the attached Figures, wherein:

FIG. 1 is a representation of the prior art wherein software providesimage resizing through cropping;

FIG. 2 is a representation of the prior art where image resizing isbased upon a non-linear scaling;

FIG. 3 depicts image resizing according to the prior art of S-H Lee inUS Patent application 2008/0,019,439;

FIG. 4 depicts image resizing according to the prior art of H. Chao inUS Patent application 2008/0,095,470;

FIG. 5 depicts content aware image resizing according to the prior artof S. Aviden in U.S. Pat. No. 7,477,800;

FIG. 6 depicts an embodiment of the invention depicting generation ofthe first gradient maps and associated reduced second gradient maps withtheir reuse in repeated removal of selected pixel paths to resize animage;

FIG. 7 A depicts an embodiment of the invention in establishing a pixelpath within a gradient map;

FIG. 7B depicts an embodiment of the invention in establishing a pixelpath within a gradient map;

FIG. 7C depicts repeated pixel path determinations made upon the reducedsecond gradient map according to an embodiment of the invention forreduced processing complexity and improved speed;

FIG. 7D depicts a process flow wherein repeated pixel pathdeterminations are made upon the second saliency map according to anembodiment of the invention for reduced processing complexity andimproved speed;

FIG. 8 depicts an embodiment of the invention wherein pixel pathselection is determined from different second reduced gradient mapswhich are derived from a common first gradient map;

FIG. 9 depicts an embodiment of the invention wherein pixel pathselection is made within a second reduced gradient map and interpolatedfor image adjustment during image resizing;

FIG. 10 depicts a limitation within the prior art of S. Aviden in U.S.Pat. No. 7,477,800 wherein seam carving removes pixels with significantimage content;

FIG. 11 depicts an embodiment of the invention within an authoringenvironment wherein significant image content is protected fromselection in the pixel path determinations for image resizing;

FIG. 12 depicts results of prior art linear scaling and an embodiment ofthe invention wherein a portion of a person's body is protected duringthe image resizing;

FIG. 13 depicts an embodiment of the invention within an authoringenvironment wherein image content is preferentially selected in thepixel path determinations for image resizing;

FIG. 14 depicts an embodiment of the invention wherein within anauthoring environment image content is identified by the user as beingpreferentially removed and protected in the pixel path determinationsand image resizing;

FIG. 15 depicts a process flow according to an embodiment of theinvention wherein pixel path determination is executed upon a portabledevice in dependence upon characteristics of the portable device;

FIG. 16 depicts multiple pixel selection within image resizing accordingto an embodiment of the invention based upon the reduced second gradientmap;

FIG. 17 depicts an embodiment of the invention wherein pixel pathdetermination based upon the reduced second gradient map is performedwithin a video authoring or display environment; and

FIG. 18 depicts protection of image content from selection during pixelpath determination within image resizing according to an embodiment ofthe invention.

DETAILED DESCRIPTION

The present invention is directed to content aware resizing ofaudiovisual and image content.

Reference may be made below to specific elements, numbered in accordancewith the attached figures. The discussion below should be taken to beexemplary in nature, and not as limiting of the scope of the presentinvention. The scope of the present invention is defined in the claims,and should not be considered as limited by the implementation detailsdescribed below, which as one skilled in the art will appreciate, can bemodified by replacing elements with equivalent functional elements.

Reference below is made in respect of FIGS. 6 through 14 and FIGS. 16through 18 to an authoring environment in respect to the discussion,such as for example a desktop publishing environment. The scope of thepresent invention should not be considered as limited by theseimplementation details, as one skilled in the art will appreciate, whichcan be modified such that embodiments of the invention may operate withor without user intervention or may be employed in display andpresentation environments to a user, such as described in FIG. 15.

Further in FIG. 15 reference is made to a portable device in thedetermination of the parameters in establishing aspects of the resizingoperation which extend beyond the intended image size. The scope of thepresent invention should not be considered as limited by theseapplication details, as one skilled in the art will appreciate, whichcan be varied according to the particular portable device but also applyto the wider range of devices upon which user activities may requirecontent aware image resizing.

Within the background to the invention discussed supra descriptions ofFIGS. 1 through 5 have been included and are not repeated here.

Referring to FIG. 6 there is depicted an exemplary flow according to anembodiment of the invention. As shown a source image 610 is provided forwhich a resizing operation is required within an authoring environment,the authoring environment omitted for clarity. The content awareresizing process then generates first horizontal saliency map 620 andfirst vertical saliency map 625 which represent the horizontal andvertical saliencies within the image which are determined from Equations3 and 4 below:

Saliency_(horizontal)(n _(ij))=|I(n _(i,j+1))|−|I ₍ n _(i,j−1))|  (3)

Saliency_(vertical)(n _(i,j))=|I(n _(i+1,j))−|(I(n _(i−1,j))|  (4)

where I(n_(i,j)) is the intensity of the i^(th), j^(th) pixel in theimage.

Each of the first horizontal saliency map 620 and first verticalsaliency map 625 are then scaled to generate second reduced horizontalsaliency map 630 and second reduced vertical saliency map 635. These arethen employed to generate the cost functions for removing a pixel seamin each of the horizontal and vertical directions. A selected verticalseam from second reduced horizontal saliency 630 is shown as pixel path645 projected onto resizing image 640. Removal of the pixels identifiedby pixel path 645 would reduce the horizontal dimension of the sourceimage 610. Alternatively insertion of replica pixels identified by pixelpath 645 would increase the horizontal dimension. Accordingly the sourceimage 610 is scaled based upon a pixel path that is determined throughthe scaling transformation in respect of the horizontal and verticalsaliencies defined in Equations (3) and (4) supra.

Referring to FIG. 7 A there is depicted a process flow 700A according toan embodiment of the invention in establishing a pixel path within areduced saliency map. The process starts with first pixel map 710A ofdimension 5×3, which represents a subset of a reduced saliency map suchas second reduced horizontal saliency map 630 or second reduced verticalsaliency map 635 in FIG. 6 supra. The process then determines theinterconnected paths between the pixels on the first row and the secondthat are connected, resulting in second pixel map 720A which shows thisconnectivity between the first row and second row such that the processthen sums these paths giving the middle summation in third pixel map730A together with the mapping of connectivity between the summed secondrow and third row. The resultant summation being shown in fourth pixelmap 735A along with the connectivity paths from each row to the next.According to an embodiment of the invention process flow 700A is set todetect the minimum summation in the pixel path and thereby determinesthis is in the summed path provided in fifth pixel map 740A. Accordinglythe pixels within the subset of the saliency map are selected asdepicted by sixth pixel map 745A. In the final step the process removesthese pixels thereby generating seventh pixel map 750A which is now ofdimension 4×3. In the process according to the embodiment of theinvention this pixel removal in the reduced saliency map follows removalof pixels within the audiovisual content, such as described below inrespect of FIG. 15.

It would evident to one skilled in the art that process flow 700A doesnot take into account the pixels removed from the saliency map such asis evident in the comparison of sixth and seventh pixel maps 745A and750A respectively where simply the pixel path selected has been removed.In other embodiments of the invention, for instance where a portion ofthe saliency map has a localized reduction in saliency compared with theoverall saliency map the reduction algorithm may perform some form ofcompensation such as shown below in Table 1. As shown on the left isseventh pixel map 750A according to process flow 700 in FIG. 7. On theright is a compensated pixel map representing the same pixel pathremoval but where now pixels adjacent the removed pixel arere-calculated according to Equations SA and SB below:

S ^(k+l)(i−1,j)=S ^(k)(i−1,j)+S ^(k)(i,j)/2  (5A)

S ^(k+1)(i−1,j)=S ^(k)(i+1,j)+S ^(k)(i,j)/2  (5B)

where S^(k) (i, j) represents the saliency value at the i^(th), j^(th)pixel for step k in the image resizing process. It would be apparentthat similar equations as Equations 5A and 5B exist for removing ahorizontal pixel path. Such a compensated pixel map locally increasessaliency above the initially calculated values upon removal of a pixelpath which would weight a subsequent pixel path determination away fromthe same region of the saliency map such that multiple pixel pathdeterminations do not always run through the same portion of thesaliency map and hence the original image.

TABLE 1 Left: Saliency map after pixel path removal according to process700 of FIG. 7A Right: Saliency map after pixel path removal withcompensation as discussed supra 3 5 4 7 3 5 4 8 7 1 9 8 7 1 11.5 10.5 69 7 8 6 9 7 9

It would be evident to one of skill in the art that the selected pathwithin process 700A by virtue of having the lowest summation ofsaliencies represents a path of pixels that have low difference inintensity to their neighbouring pixels in a particular direction. Thesepixels are not necessarily at a minimum within the reduced saliency mapfor the other direction and hence not necessarily the same pixels aswould be selected in the process of A vidan when employed on the sameimage. As such removing these pixels from the image should notsignificantly affect the content for the user whilst allowing the imagedimension to be reduced. It would evident to one skilled in the art thatzero saliency or very low saliencies may reflect areas of consistentintensity rather than lack of content. As such regions where salienciesexceed a predetermined threshold may be subjected to a second process todetermine whether they are simply pixels reflecting low intensityvariations and hence sacrificial content or significant content ofconsistent intensity. For example the second process may be to calculateand compare a second saliency for a particular pixel, see for exampleEquations 5C and 5D below; with the first saliency such that upon aprecondition being met the calculated saliency is replaced with apredetermined value.

Saliency2_(horizontal)(n _(i,j))=|I(n _(i,j+n))|−|I(n _(i,j−n))|  (5C)

Saliency2_(horizontal)(n _(i,j))=dI(i,j)/dj  (5D)

Referring to FIG. 7B there is depicted a process flow 700B according toan embodiment of the invention in establishing a pixel path within areduced saliency map. The process starts with first pixel map 710B ofdimension 5×3, which represents a subset of a reduced saliency map suchas second reduced horizontal saliency map 630 or second reduced verticalsaliency map 635 in FIG. 6 supra. The process then determines theinterconnected paths between the pixels on the first row and the secondthat are connected, resulting in second pixel map 720B which shows thisconnectivity between the first row and second row such that the processthen sums these paths giving the middle summation in third pixel map730B together with the mapping of connectivity between the summed secondrow and third row. The resultant summation path being shown in fourthpixel map 735B along with the connectivity paths from each row to thenext. According to an embodiment of the invention process flow 700B isset to detect the maximum summation in the pixel path and therebydetermines this is in the summed path provided in fifth pixel map 740B.Accordingly the pixels within the subset of the saliency map areselected as depicted by sixth pixel map 745B. In the final step theprocess adds these pixels into the first pixel map 710B therebygenerating seventh pixel map 750B which is now of dimension 6×3.

It would be evident to one of skill in the art that the selected pathwithin process 700B by virtue of having the highest summation ofsaliencies represents a path of pixels that have high difference inintensity to their neighbouring pixels. As such replicating those pixelswithin the image that relate to those within the reduced saliency mapshould preserve the visually significant content for the user whilstallowing the image dimension to be increased.

It would be apparent to one skilled in the art that the pixel pathselection in FIGS. 7 A and 7B may be subject to additional constraintsor determined on alternative basis. For example it may be a constraintthat the pixel path originates within a predetermined distance of theimage edge such that the central image content is preserved irrespectiveof its pixel saliency summation, where the assumption is that mostsignificant content is within the central portion of the image.Alternatively a summation may be performed over predetermined regions ofthe second saliency map such that regions of higher than averageaccumulated saliency are identified and preserved. Optionally the pixelpath selection when the adjustment is a significant percentage of theoriginal image dimension may be established such that pixel paths shouldbe maximized in one direction and minimized in another. Similarly wherepixel path selection has been described as seeking a minimum/maximum theconverse of seeking the maximum/minimum for the same image resizingoperation exists. Many alternatives exist within the scope of theinvention.

Referring to FIG. 7C there is depicted a process flow 700C whereinrepeated pixel path determinations are made upon the reduced secondsaliency map according to an embodiment of the invention for reducedprocessing complexity and improved speed. As such within process 700C afirst reduced saliency map 710C is shown, equivalent for example tofirst pixel maps 710A and 710B of FIGS. 7 A and 7B respectively orpredetermined portions of second reduced horizontal saliency map 630 orsecond reduced vertical saliency map 635 in FIG. 6 supra. First reducedsaliency map 710C is a 8×5 array of reduced saliency data, being eitherthe horizontal saliency or vertical saliency of that localized region ofthe image as reduced saliency map 710C is a reduced dimensional matrixof the corresponding first saliency map, for example first horizontalsaliency map 620 or first vertical saliency map 625 as disclosed in FIG.6. As such a pixel within first reduced saliency map 710C represents Npixels, wherein N represents the scale reduction applied to thecorresponding first saliency map. Saliency S(i, j) may alternatively bedefined for example by Equations 6 and 7 below rather than by Equations3 and 4.

$\begin{matrix}{{S\left( {i,j} \right)} = {{{{I\left( {i,{j + \frac{N}{2}}} \right)} - {I\left( {i,{j - \frac{N}{2}}} \right)}}}/N}} & (6) \\{{S\left( {i,j} \right)} = {{{{I\left( {{i + \frac{N}{2}},j} \right)} - {I\left( {{i - \frac{N}{2}},j} \right)}}}/N}} & (7)\end{matrix}$

where I(i, j) represents the intensity of the ith, r pixel in the sourceimage.

In first pixel summation map 720C the summed saliency values S(i, j)from each pixel within the top row to the bottom row are shown forconnected paths. Also shown is first pixel path 725C selected from thefirst pixel summation map 720C, in this case based upon the lowest sum.The pixels within the image content being resized and first reducedsaliency map 710C corresponding to the first pixel path 725C are thenremoved resulting in second reduced saliency map 730C, i.e. pixelsS(1,4)=2, S(2,4)=1, S(3,4)=1, S(4,4)=2, and S(5,5)=5 are removed.Corresponding pixels in the image are removed that correspond to theselected pixels in first pixel path 725C thereby reducing the imagewidth based upon its content. Using second reduced saliency map 730C thesummation process is repeated and second pixel summation map 740C isgenerated. Again a pixel path 745C is established such that thecorresponding pixels within the second reduced saliency map 730C areremoved, i.e. pixels S(1,1)=1,S(2,2)=3,S(3,1)=3,S(4,1)=3, and S(5,1)=4.Again corresponding pixels in the reduced image from the previousremoval of pixels are removed, further reducing the width of the image.

Removal of the selected pixels in second reduced saliency map 730Cresults in third reduced saliency map 750C. As above the process thengenerates third pixel summation map 760C and selects the next pixel path765C. Applying the selected path to third reduced saliency map 750Cresults in fourth reduced saliency map 770C of dimensions 5×5 i.e.removing pixels S(1,3)=3, S(2,2)=3,S(3,3)=1, S(4,2)=3, and S(5,2)=5. Assuch it would be evident to one skilled in the art that the reduction ofthe image is accomplished without recalculating the reduced saliencymaps from the corresponding horizontal saliency map or vertical saliencymap, such as horizontal saliency map 630 and vertical saliency map 640in FIG. 6. As such scaling the image is achieved with a significantreduction in the processing complexity when compared with the prior artof content aware image resizing, such as S. Aviden et al who recalculatethe top level pixel maps from the resultant image after each “seam” iscarved or inserted. Such a reduction in processing complexitybeneficially provides for the pixel path methodology to be deployedwithin portable consumer electronics with reduced processingcapabilities when compared to laptop PCs with dual-core 2 GHz processorsand 4 GB RAM.

It would be apparent to one of skill in the art that the pixel pathadjustment provided within each of the image content and saliency mapsas a result of pixel path determination within the reduced saliency mapmay not always remove the corresponding number of pixels within thesehigher plane maps, such as described below in FIG. 16. It would beapparent that image resizing may require an increase/decrease in anumber of pixels that does not match an integer scaling ratio, i.e. aprime number, which requires either the saliency mapping be performedwith a scaling equal to the prime number, not be scaled, or be left at asize not matching the target. Considering simply resizing involvingbetween 1 and 1000 pixels there are 168 prime numbers. For example,removing 367 pixels may be achieved with 367 single pixel path removalswhich is time consuming but leads to the desired result.

Alternatively as described in embodiments of the invention the scalingprovides an increased speed, for example 183 removals of 2 pixel widepaths, 92 removals of 4 pixel wide paths, 61 removals of 6 pixel widepaths, or 37 removals of 10 pixel wide paths. In all cases the finalimage is at the incorrect final dimension. Accordingly it would beapparent that providing the process with the ability to removal a numberof pixels within the image content that does not match the scalingallows the final image to be scaled in a content aware manner to thecorrect final dimension. Accordingly, 36 removals of 10 pixel wide pathwith a ÷10 scaling may be followed by a final 7 pixel wide leaves theimage at the target resize dimension. Similarly applying 36 removals of6 pixel wide paths followed by a final single wide pixel path.Accordingly the process may dynamically select a scaling to meet therequirements for speed and processing whilst achieving the final targetdimension.

Referring to FIG. 7D there is depicted a process flow 700D whereinrepeated pixel path determinations are made upon the second saliency mapaccording to an embodiment of the invention for reduced processingcomplexity and improved speed. As such within process 700D a firstreduced saliency map 710D is shown, equivalent for example to firstpixel maps 710A and 710B of FIGS. 7 A and 7B respectively orpredetermined portions of second reduced horizontal saliency map 630 orsecond reduced vertical saliency map 635 in FIG. 6 supra. First reducedsaliency map 710D is a 8×5 array of reduced saliency data, being eitherthe horizontal saliency or vertical saliency of that localized region ofthe image as reduced saliency map 710C is a reduced dimensional matrixof the corresponding first saliency map, for example first horizontalsaliency map 620 or first vertical saliency map 625 as disclosed in FIG.6. As such a pixel within first reduced saliency map 710D representseffectively N pixels, wherein N represents the scale reduction appliedto the corresponding first saliency map.

In first pixel summation map 720D the summed saliency values S(i, j)from each pixel within the top row to the bottom row are shown forconnected paths. Also shown is first pixel path 725D selected from thefirst pixel summation map 720D, in this case based upon the lowest sum.The pixels within the saliency map, not shown for clarity but being thatfrom which first reduced map 710D was derived, corresponding to thefirst pixel path 725D are then removed. The resulting saliency map, alsonow shown for clarity, is then reduced to yield second reduced saliencymap 730D, of dimensions 7×5, which whilst globally similar to firstreduced saliency map 710D as only a portion of the pixels were removeddiffers in those pixels identified by region 735D, i.e. pixelsS(1,4)=4,S(2,4)=6, and S(3,4)=2. As discussed supra the correspondingpixels in the image were also removed in addition to those within thesaliency map corresponding to the selected pixels in first pixel path725D thereby not only reducing the image width but doing so based uponits content. The process flow 700D then uses second reduced saliency map730D to repeat the summation process from which second pixel summationmap 740D is generated. Again a pixel path 745D is established based uponthe minimum saliency summation and the process flow 700D then removescorresponding pixels within both the image and saliency map. From thisresulting modified saliency map, not shown for clarity process flow 700Dcalculates the third reduced saliency map 750D.

Third reduced saliency map 750D of dimensions 6×5 is again globallysimilar to second reduced saliency map 730D, as only a portion of thepixels within the saliency map were removed which forms the source ofthird reduced saliency map 750D, but differs in region 755D whichdiffers now in S(3,1)=6,S(4,1)=5, and S(5,1)=6. Again process flow 700Dperforms another summation process resulting in third pixel summationmap 760D and selects the next pixel path 765D having lowest saliencysummation. Applying this selected path to both the image and saliencymap as discussed supra further reduces the image width based upon itscontent and results in a new saliency map, not shown for clarity, fromwhich a fourth reduced saliency map 770D, now of dimensions 5×5 isgenerated. As the dimensions of the reduced saliency map reduces theregion that differs from the preceding reduced saliency map increasestypically. As such, now region 775D now differs in

S(1,3)=5,S(1,4)=6,S(2,3)=7,S(2,4)=7,S(3,3)=4,S(3,4)=5,S(4,2)=4,S(4,3)=5,and G(5,3)=7

As such it would be evident to one skilled in the art that the reductionof the image is accomplished according to the embodiment of theinvention presented in FIG. 7D without recalculating the saliency mapsfrom the corresponding image. However, unlike the preceding embodimentin FIG. 7C the reduced saliency maps are calculated from the applicablehorizontal saliency map or vertical saliency map, such as horizontalsaliency map 620 and vertical saliency map 625 in FIG. 6, which isreduced during the process. As such scaling the image is achieved with asignificant reduction in the processing complexity when compared withthe prior art of content aware image resizing, such as S. Aviden et alwho recalculate the top level pixel maps from the resultant image aftereach “seam” is carved or inserted.

Optionally the pixel path selected is based upon multiple conditions.For example, the pixel path selected is not only one meeting a minimumsummation or a maximum summation such as presented supra in respect ofFIGS. 7 A and 7B but is one where the pixel path is one with a lowsummation and results in the minimum change in an overall measure of thereduced saliency map for example.

Considering portable devices today with significant market share withintheir respective markets such as Research in Motion's popular Blackberry8100, 8300 and 8700 series cellular telephones employing an Intel PXA901processor at 312 MHz with 16 MB RAM, Nintendo's DSi handheld gameconsole employs two ARM processors, an ARM9E processor operating at 133MHz and an ARM7TDMI coprocessor operating at 33 MHz, with the ARM9Eprocessor controlling game play and image processing, and Apple's iPodportable audiovisual media players series including the Nano and 40which employ dual 80 MHz ARM 7TDMI processors. All of these devicessupport Internet access and hence would benefit from dynamic imageprocessing when browsing the Internet as their capabilities areincreased. As such embodiments of the invention support use withinportable consumer devices to dynamically resize image with content awarescaling in real-time thereby allowing them to access any publishedaudiovisual or image content already in existence without requiringpreprocessing by desktop publishing software suites and increased filesizes to handle the header embedded seam carving sequence such as taughtby S. Aviden.

It would be evident to one skilled in the art that the path selectionstep resulting in third pixel path 765C could have selected from fourpotential paths,

S(1,3)→S(2,2)→S(3,3)→S(4,2)→S(5,1);S(1,3)→S(2,2)→S(3,3)→S(4,2)→S(5,2);S(1,3)→S(2,2)→S(3,3)→S(4,3)→S(5,2);S(1,3)→S(2,2)→S(3,3)→S(4,2)→S(5,4).

Optionally the pixel path content aware image resizing process may havesecondary routing protocols that establish which of these to selectpreferentially. For example the secondary protocol may be to avoidvertical pixel combinations wherever possible, thereby removingS(1,3)→S(2,2)→S(3,3)→S(4,3)→S(5,2) as an option, or seeks to removepixels at the edge of the image thereby favoringS(1,3)→S(2,2)→S(3,3)→S(4,2)→S(5,1).

Referring to FIG. 8 there is depicted according to an embodiment of theinvention image process flow 800 wherein pixel path selection isdetermined from one of two different second reduced saliency maps, beingfirst and second reduced saliency maps 820 and 830 respectively, whereineach second saliency map is derived from a common first saliency map810. According a source image 805 provides the pixel intensity arrayI(i, j) that acts as the source data for calculatingSaliency_(horizontal)(n_(i,j)) and Saliency_(vertical) which form thebasis of horizontal saliency map 810A and vertical saliency map 810B.This step in the process flow being common to two users, one on a laptopcomputer 860 and another on a cellular telephone 870. The process inexecution upon the laptop computer 860 generates a first pair of reducedsaliency maps 830 which are then used to generate dynamically scaledfirst and second resized images 840 and 850 as the user adjusts theonscreen dimensions of a web browser whose content includes the sourceimage 805.

In contrast the process in execution upon a cellular telephone 870generates a second pair of reduced saliency images 820 that are thenused to generate third resized image 880. Accordingly the process runson the two different devices in a manner that adjusts to suit the deviceupon which it is executing. It would be evident to one skilled in theart that a resizing operation geared to a 240×320 pixel 2.1″ cellulartelephone 870 display has different requirements to one displayingimages upon a 17″ 1920×1080 display on a laptop computer 860. As aresult the process according to embodiments of the invention allows forcontent aware image resizing that is configurable to the device uponwhich the process is operating. This configurable processing is notcontained within the prior art content aware resizing approachesdiscussed supra.

Now referring to FIG. 9 there is depicted a flow 900 according to anembodiment of the invention wherein pixel path selection is made withina second reduced saliency map and interpolated for image adjustmentduring image resizing. As such there is shown a source image 910 uponwhich a resizing operation is to be performed, the intensity data I(i,j) of which is employed in generating first saliency map 920 from whichsecond reduced saliency map 930 is generated. The second reducedsaliency map 930 is then the data source for the pixel pathdetermination process, such as presented supra in respect of FIGS. 7 A,7B and 7C. A pixel path portion 940 of the determined pixel path 935from second reduced saliency map 930 is shown comprising a 4×4 matrixwith selected pixels 945 infilled. Within this example scaling betweenfirst saliency map 920 and second reduced saliency map 930 is a factorof 3. As such pixel path portion 940 is scaled back by a factor of 3 togenerate expanded pixel path 950 within which selected pixels 945 areshown as highlighted pixels 955.

Next flow 900 executes an interpolation process to generate interpolatedpixel map 960 wherein the selected pixels 955 are shown together withinterpolated pixels 964. Next each selected pixel 955 and interpolatedpixel 964 are replaced by pixel path element 972 which are determined asthe average of each neighbouring pixel 974, i.e. P(i, j)=(I(i−1,j)+I(i+1, j))/2. The pixel path elements 972 are then inserted into theoriginal image 910 to generate resized image 980. It would be evidentthat within FIG. 9 the flow 900 described relates to an increase inimage dimensions as opposed to a reduction. Accordingly the processdescribed in FIGS. 7C and 7D supra for selecting sequential paths andremoving them to reduce a dimension may be applied in reverse andmultiple pixel paths inserted into the image. Accordingly rather thanthe saliency maps and reduced salience maps decreasing in dimension theywould increase. It would evident to one skilled in the art thatgeneration of pixel path elements 972 may be varied, such as for examplerather than using the average of neighbouring pixels the value insertedis that representing the pixel with the minimum value between theneighbouring pixels 974 and interpolated pixel 964.

Now referring to FIG. 10 there is depicted a limitation within the priorart of S. Aviden in U.S. Pat. No. 7,477,800 wherein seam carving removespixels with significant image content. As shown a source image 1010 ispresented that contains a first region 1015 of very little variation,being an item of clothing for one of the two individuals within thesource image 1010. The prior art of S. Aviden was employed by W. Wedlerfor this source image 1010 (see Image Resizing by Seam Carving—Project2—Computational Photography at Carnegie Mellon University,http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj2/www/wwedler).Shown in second image 1020 are multiple seams 1025 determined for animage reduction process wherein a majority of the multiple seams 1025run through the first region 1015 As a result when these seams areremoved to generate resized image 1030 the first region 1015 is removedpreferentially resulting in second region 1035 which has essentiallyremoved the majority of the torso of the individual within the image. Asdiscussed supra in respect of FIG. 7 A an automated resizing processupon a device may having generated a first saliency map or secondreduced saliency map according to the invention have identified that asubstantial region within the map that had low saliency, namely firstregion 1015, such that pixel paths would preferentially pass through it,for example by comparing saliencies calculated using for exampleEquation (3) with either Equation (5C) or (5D), or through anotherprocess. In these circumstances either replacing saliencies with apredetermined value such that these pixels were not preferentiallyselected or removing paths calculated through these pixels would resultin retention of such a region.

Within a desktop publishing application such a restriction may be madeusing a mask applied to the second reduced saliency map from which thepixel paths are selected. Such an approach according to an embodiment ofthe invention within an authoring environment is shown in FIG. 11wherein there is depicted a process flow 1100 establishing a pixel pathwithin a saliency map, subsequently referred to as pixel maps. Theprocess starts with first pixel map 1110 of dimension 5×3, whichrepresents a subset of a saliency map such as second reduced horizontalsaliency map 630 or second reduced vertical saliency map 635 in FIG. 6supra for example. The process then determines the interconnected pathsbetween the pixels on the first row and the second, resulting in secondpixel map 1120 which shows this connectivity between the first row andsecond row. However, S(1,5)=|I(i, j+1)−I(i, j−1)|=2 for example, hasbeen masked, shown by hatching in that cell in first and second pixelmaps 1110 and 1120 respectively. As such the connectivity mappingbetween the first and second rows does not include S(2,5)→S(1,5) suchthat when the process sums these paths giving the middle summation inthird pixel map 1130 this path is not calculated or mapped. Third pixelmap 1130 also showing connectivity mapping between the summed second rowand third row. The resultant summation path for the 5×3 array beingshown in fourth pixel map 1135 along with the connectivity paths fromeach row to the next. According to an embodiment of the inventionprocess flow 1100 is set to detect the minimum summation in the pixelpath and thereby determines this is in the summed path shown in fifthpixel map 1140.

The selected path as shown in fourth pixel map 1140 beingS(1,1)→S(2,2)→S(3,1) whereas in FIG. 7 A supra using the same pixel map,without the masking applied to S(1,5), the path selected wasS(1,5)→S(2,4)→S(3,5). Accordingly the pixels within the subset of thesaliency map are selected as depicted by sixth pixel map 1145 which arethen removed by the process to generate seventh pixel map 1150 which isnow of dimension 4×3 with S(1,5)=2 still protected for subsequent pixelmap operations. It would be evident that rather than limiting theconnectivity mapping aspect of the process flow that alternatively thesaliency value stored may be replaced with a saliency value that wouldremove the pixel from summed routes. For example where the pixel pathprocess seeks a minimum summation making the protected pixels have highsaliency would remove then from the pixel path selection, similarlywhere the pixel path process seeks a maximum summation making theprotected pixels have low saliency would remove then from the pixel pathselection. Other options would be apparent to one of skill in the art.

Referring to FIG. 12 there are depicted the results of prior art linearscaled 1220 and an embodiment of the invention in content aware scaledimage 1230 as applied to an original image 1210. In linear scaled 1220the woman's face is distorted whereas by protecting this portion 1205 ofthe original image 1210 the content aware scaled image 1230 has a womanwith a longer body as desired but with a natural head proportion.

In other authoring applications it may be appropriate to remove contentpreferentially. Such a process 1300 is depicted in FIG. 13 according toan embodiment of the invention. The process starts with first pixel map1310 of dimension 5×3, which represents a subset of a saliency map suchas second reduced horizontal saliency map 630 or second reduced verticalsaliency map 635 in FIG. 6 supra for example. The process thendetermines the interconnected paths between the pixels on the first rowand the second that are connected, resulting in second pixel map 1320which shows this connectivity between the first row and second row.However, whilst connectivity S(2,2)→S(1,1) represents a lower summationthan S(2,2)→S(1,2) the process 1300 forces this connectivity so thatpixel S(1,2) is contained within the calculated summations. S(1,2)=|I(i,j+1)−I(i, j−1)|=5 for example, has been masked, shown by shading in thatcell in first and second pixel maps 1310 and 1320 respectively. As suchthe connectivity mapping continues to third pixel map 1330 showingconnectivity mapping between the summed second row and third row. Theresultant summation path for the 5×3 array being shown in fourth pixelmap 1335 along with the connectivity paths from each row to the next.According to an embodiment of the invention process flow 1300 is set todetect the minimum summation in the pixel path and thereby determinesthis is in the summed path provided in fifth pixel map 1340.

The selected path as shown in fourth pixel map 1340 beingS(1,2)→S(2,2)→S(3,1) whereas in FIG. 7 A supra using the same pixel mapwithout the masking to S(1,S) being applied the path selected wasS(1,S)→S(2,4)→S(3,5). Accordingly the pixels within the subset of thesaliency map are selected as depicted by sixth pixel map 1345 which arethen removed by the process to generate seventh pixel map 1350. It wouldbe evident that rather than limiting the connectivity mapping aspect ofthe process flow that alternatively the saliency value stored may bereplaced with a saliency value that would removes the pixel from summedroutes. For example where the pixel path process seeks a minimumsummation making the preferred pixels have low saliency, i.e. zero,would preferentially weight to these pixels in pixel path selection,similarly where the pixel path process seeks a maximum summation makingthe protected pixels have high saliency would remove then from the pixelpath selection. Other options would be apparent to one of skill in theart. Such options may in some circumstances force the pixel pathselection to these pixels even when local pixel paths may have hadsummations that previously weighted path selection to them.

Now referring to FIG. 14 there is depicted an embodiment of theinvention wherein within an authoring environment image content within asource image 1410 is identified by the user as being both preferentiallyremoved and protected in the pixel path determinations and imageresizing. Accordingly in first image 1420 the user has selected the farleft individual for removal with first removal mask 1422, but beingconscious of the middle left individual and the background tower hasprotected these with first and second protection masks 1424 and 1426respectively. Then applying a content aware image resizing processaccording to an embodiment of the invention yields first output image1430 wherein the selected individual has been removed but the overallcontent has minimal artifacts to indicate to a viewer that the image wasprocessed.

An alternate authoring is shown in second image 1440 where the user hasselected the far right individual for removal with second removal mask1442, but being conscious of the middle right individual and thebackground building has protected these with third and fourth protectionmasks 1424 and 1426 respectively. Then applying a content aware imageresizing process according to an embodiment of the invention yieldssecond output image 1450 wherein the selected individual has beenremoved but the overall content has minimal artifacts to indicate to aviewer that the image was processed.

It was noted supra that a content aware image resizing process accordingto embodiments of the invention may be deployed within a range ofelectronic devices including portable devices allowing the process toresize images retrieved by users rather than requiring all images theyaccess be authored in a suite providing header encoded seam carvingsequences such as taught within the prior art by S. Aviden. Referring toFIG. 15 there is depicted a process flow 1500 according to an embodimentof the invention wherein pixel path determination for content awareimage resizing is executed upon a portable device in dependence uponcharacteristics of the portable device. As such the process begins atstep 1502 where the user opens a web browser interface, or accesses theInternet and retrieves a web page through a specific Internet accessapplication such as the browsers within Blackberry and iPhone PDAsrather than Windows Internet Explorer, Mozilla, etc. As such in step1504 they access a web page and as part of that digital content relatingto an image is downloaded in step 1506. The application in executionupon the user's electronic device establishes the display dimensions forthe downloaded image in step 1508 and then in step 1510 retrieves devicesettings relating to the portable device the user is using, not shownfor clarity.

Subsequently in step 1512 the image scaling ratio required for the imageis determined and then, based upon the device settings and image,scaling the scaling ratio of the reduced saliency pixel map isdetermined in step 1514. Next in step 1516 the horizontal saliency map1H is generated, and subsequently in step 1518 the vertical saliency mapIV is calculated. These together with the scaling ratio of the saliencymaps determined in step 1514 are used to calculate horizontal reducedsaliency map 2H and vertical reduced saliency map 2V in steps 1520 and1522. In step 1524 a counter is set, X=1, and in step 1526 applicablepixel paths within reduced saliency horizontal and vertical maps 2H(X)and 2V(X) respectively are determined. Next in step 1528 these pixelpaths are scaled as appropriate, such as discussed supra in respect ofFIG. 9 and then an interpolation is performed in step 1530 to establishthe applicable horizontal and/or vertical seams. In step 1532 theseinterpolated pixels are replaced by “proper” pixels which are generatedusing the neighboring pixels according to a predetermined algorithm.

This determined pixel seam is then applied to the image in step 1534 andthe pixel path is then applied to the saliency maps 1H(X) and 1 V(X) asappropriate in step 1536. Then in step 1538 the process determineswhether the image size required has been achieved, which if it hasresults in the process moves to step 1542 and terminating. If furtherresizing is required the process moves to step 1540, increments thecounter, X=X+1, and loops back to step 1520 so that the process cancontinue such as described for example in respect of FIG. 10, which asoutlined allows multiple pixel path selection without recalculation ofthe saliency energy map such as outlined supra.

It would be evident to one skilled in the art that the characteristicsof the portable device retrieved in the process flow and impacting thecontent aware resizing process may be other than display dimensions andmay include but not be limited to processor speed, processor loadingwith other applications, graphics display driver settings, and batterystatus. For example, a low resolution display combined with a lowprocessor speed may result in employing a high scaling ratio betweensaliency map and reduced saliency map whilst high resolution display andhigh processor speed may typically employ a low scaling ratio unless thebattery status is of a low battery wherein minimizing processing maybecome more important such that a high scaling ratio IS again employed.Other combinations and eventualities would be evident to one of skill inthe art.

It would be apparent that under some circumstances it would be desirableto perform the pixel path based content aware resizing in a manner thatis less precise or faster than described in respect of embodimentspresented supra in respect of FIGS. 6 through 15. Referring to FIG. 16there is depicted a process 1600 wherein pixel path determination ismade upon a reduced second saliency map according to an embodiment ofthe invention which is a variant of FIG. 9 and provides reducedprocessing complexity and improved speed. Hence, as with the supraembodiments a source image 1610 is initially converted to a firstsaliency map 1620 which is then scaled, by a factor N, to providereduced saliency map 1630. The embodiment in FIG. 16 does notspecifically address horizontal and vertical versions of the firstsaliency map 1620 and reduced saliency map 1630 for simplicity.Accordingly as presented supra in respect of FIG. 9 the processdetermines a pixel path 1640 comprising pixels 1645, but now ingenerating scaled pixel path 1650 rather than discrete pixels beingselected and the path interpolated the scaled pixel path has N×N pixelsselected as groups 1655, where N was the scaling ratio applied to thefirst saliency map, such that the pixel path is N pixels wide andcontinuous across the image. As such a single pixel path removal stepremoves N pixels in either the horizontal or vertical direction therebyreducing the processing by a factor of N. It would evident to oneskilled in the art that the factor N as discussed supra in respect ofFIG. 8 may be dynamically determined based upon static characteristicsof the device but also optionally dynamic aspects of the device such asprocessor load and battery status for example.

Within the embodiments presented supra the consideration has been todigital content that relates to images and hence of a static contenttemporally unless resized by the activities of the user. However, itwould be evident that the digital content accessed by users may includeadditionally audiovisual content such as downloaded or streamedaccording to international video standards such as Audio VideoInterleave (AVI), Movie Picture Experts Group (MPEG, e.g. mp4), andWindows Media Video (WMV). Referring to FIG. 17 there is depicted aprocess 1700 relating to multiple pixel path selection for content awareimage resizing of audiovisual data. Hence there is shown an audiovisualsequence 1710 comprising a series of “frames” 1710A through 1710N. Asfirst “frame” 1710A is received it is converted to first saliency map1720A which is then converted to first reduced saliency map 1730A asdiscussed supra in respect to other embodiments of the invention, andthen the pixel path(s) is/are selected as shown in first path map 1740A.Such a sequence may be repeated for each “frame” such as shown forN^(th) frame 1710N wherein the Nth saliency map 1720N is generated,converted to N^(th) reduced saliency map 1730N resulting in Nth path map1740A.

Such a process 1700 may exploit any of the adaptations identified withinthe preceding embodiments of the invention in FIGS. 6 through 16 toadapt to the scenario of audiovisual content presentation and/orauthoring. Optionally the same reduced saliency map may be applied forseveral “frames” to reduce processing complexity. It would be apparentthat potentially allowing the content aware resizing to operateindependently upon each “frame” may result in perceivablediscontinuities. As such automated dynamic masking forprotection/deletion of elements of the image such as discussed supra inrespect of FIGS. 11 through 14 may be considered. Such an automatedprocessing for example being based upon recognizing an approximaterepetitive feature in the saliency map or reduced saliency maps.Alternatively preference within a pixel path determination of asubsequent “frame” is weighted according to previous pixel paths. Suchan approach being illustrated in FIG. 18 where a first “frame” 1820through generation of a first saliency map 1820 results in the selectionof a first pixel path 1835 within first reduced saliency map 1830.Processing of a subsequent “frame” 1840 through second saliency map 1850and second reduced saliency map 1850 results in identification of secondand third pixel paths 1862 and 1864 respectively. However, process 1800applies a weighting to each of the second and third pixel paths which inthis embodiment is determined pixel path 1835. As shown second pixelpath 1862 differs in 2 pixels selected but third pixel path 1864 differsin 8. Hence, the weighting for second pixel path 1862 would be higher asit matches more closely to first pixel path 1835 thereby lending to areduction in visual discontinuities perceived by the viewer.

It would be apparent to one skilled in the art that the embodimentspresented supra have typically been described with an initial generationof a first saliency map and then the generation of a reduced saliencymap. Alternatively the reduced saliency map may be generated without thestorage or maintenance of the first saliency map. It would also beapparent that the scale between first saliency map and reduced saliencyenergy map has been presented as a constant within the above-describedembodiments. Optionally the scale may be varied across the image, suchnon-linear scaling being optionally predetermined or established independence upon characteristics of the device displaying the image orcontent of the image. Alternatively the scaling may be varied betweenthe vertical and horizontal directions of the image.

In the above embodiments recalculation of the saliency map has beenpresented as occurring at the initialization of the process and thatsubsequently reduced saliency maps are employed in determining the pixelpaths. It would be apparent to one skilled in the art that substantialimage resizing may make it beneficial to perform a recalculation of thesaliency map at a predetermined point in the process; this mayoptionally be a number of pixel seam adjustments or a percentage of theimage adjustment for example. In the above embodiments discussion withrespect to a particular format are for discussion purposes only as theembodiments are applicable to audiovisual content in multiple formatsand multiple standards.

In the above embodiments where adjustment of the process has beenpresented this has been considered primarily from the perspective ofadjusting the process in dependence upon characteristics of the deviceupon which it is being executed. Optionally the process may be adjustedin respect to the audiovisual content itself, for example a differentscaling process may be applied to JPEG files than is applied to TIFFfiles.

In the above embodiments the process has been described by considerationof different saliency maps and reduced saliency maps for the horizontaland vertical aspects of the image resizing. It would be evident to oneskilled in the art that the process may alternatively be performed withsingle reduced saliency “maps” (i.e. a three-dimensional arrays forexample) wherein each pixel within each reduced saliency map for exampleis a different plan, i.e. G(i, j, k) such that for example k=1represents the horizontal reduced saliency map and k=2 the verticalreduced saliency map. It would be evident that such an approach may beextended such that additional planes denoted by k relate to alternatesaliency calculations, masking data for protection of content, maskingdata for denoting content to remove etc.

The above-described embodiments of the present invention are intended tobe examples only. Alterations, modifications and variations may beeffected to the particular embodiments by those of skill in the artwithout departing from the scope of the invention, which is definedsolely by the claims appended hereto.

What is claimed is:
 1. A computer implementable method comprising: (a)generating at least one saliency map of a plurality of saliency maps,the at least one saliency map characterized by at least a number ofpixels equal to a number of pixels within a source audiovisual contentand each pixel within the at least one saliency map having a valueassociated with it which is determined in dependence upon determining atleast a saliency for the corresponding pixel within the sourceaudiovisual content; (b) generating at least one reduced saliency map ofa plurality of reduced saliency maps, each reduced saliency mapassociated with a saliency map and generated by applying at least onepredetermined scaling factor to the at least one saliency map; (c)applying a path determination process to the at least one reducedsaliency map, the path determination process for identifying a sequenceof pixels within the at least one reduced saliency map that meet apredetermined condition; and (d) modifying the source audiovisualcontent in dependence upon at least the sequence of pixels.
 2. Acomputer implementable method according to claim 1 wherein, the at leastone saliency map relates to at least one of a horizontal saliency and avertical saliency for the pixels within the source audiovisual content.3. A computer implementable method according to claim 1 wherein, thepredetermined condition comprises at least one of a condition relatingto a measure of the sequence of pixels and a condition relating to ameasure of the reduced saliency map.
 4. A computer implementable methodaccording to claim 1 further comprising: (e) modifying the at least onereduced saliency map in dependence upon at least the sequence of pixels;and (f) repeating steps (c), (d) and (e) until a predetermined imagecondition is met, the predetermined image condition relating to a targetfinal dimension of the source audiovisual content.
 5. A computerimplementable method according to claim 1 wherein, step (d) comprisesthe steps: (i) scaling the sequence of pixels by a predetermined factor;(ii) interpolating a path of pixels including the scaled sequence ofpixels; and (iii) at least one of removing the interpolated path ofpixels from the source audiovisual content and inserting a sequence ofproper pixels into the source audiovisual image, the sequence of properpixels comprising the interpolated path of pixels wherein each pixel inthe interpolated path is replaced by a proper pixel determined independence upon a process and at least one of the at least one saliencymap and the source audiovisual content.
 6. A computer implementablemethod according to claim 1 wherein, modifying the source audiovisualcontent comprises at least one of inserting and removing a predeterminednumber of pixels along a path within the source audiovisual content, thepath determined in dependence upon at least the sequence of pixels.
 7. Acomputer implementable method according to claim 1 wherein, the at leastone predetermined scaling factor is determined in dependence upon atleast a characteristic of at least one of the device upon which thecomputer implementable method is in execution and the source audiovisualcontent.
 8. A computer implementable method according to claim 7wherein, when the at least one is the device then the characteristic isselected from the group comprising a measure of processor loading,processor type, processor speed, accessible memory, status, displaydimension, display resolution, graphics processor, and battery life andwhen the at least one is the source audiovisual content thecharacteristic is selected from the group comprising a file format, ameasure of anticipated duration of the image on the display, and imagedimensions.
 9. A computer implementable method according to claim 1wherein, the saliency for the corresponding pixel within the sourceaudiovisual content is determined upon a component of a color spacedefining the pixel within the source audiovisual content.
 10. A computerimplementable method comprising: (a) generating a saliency mapcharacterized by at least a number of pixels equal to a number of pixelswithin a source audiovisual content and each pixel within the saliencymap having at least two values associated with it, one value determinedin dependence upon at least a saliency for the corresponding pixelwithin the source audiovisual content along a first axis of the imageand the other value determined in dependence upon at least a saliencyfor the corresponding pixel within the source audiovisual content alonga second axis of the image; (b) generating a reduced saliency map byapplying at least one predetermined scaling factor to the saliency map,each pixel with the reduced saliency map having at least first datagenerated independence upon at least the one value of a pixel within thesaliency map associated with the pixel in the saliency reduced map nadsecond data generated in dependence upon at least the other value of apixel within the saliency map associated with the pixel in the saliencyreduced map; (c) applying a path determination process to at least oneof the first data and the second data within the reduced saliency map,the path determination process for identifying a sequence of pixelswithin the reduced saliency map that meet a predetermined condition; and(d) modifying the source audiovisual content in dependence upon at leastthe sequence of pixels.
 11. A computer implementable method according toclaim 10 wherein, the one value is determined accordingSaliency(n_(i,j))=|I(n_(i,j+1))|−|I(n_(i,j−1))| and the other value isdetermined according to Saliency(n_(i,j))=|I(n_(i+1,j))|−|(n_(i−1,j))wherein I(n_(i,j)) is a particular pixel within the source audiovisualcontent.
 12. A computer implementable method according to claim 10wherein, the predetermined condition comprises at least one of acondition relating to a measure of the sequence of pixels and acondition relating to a measure of the reduced saliency map.
 13. Acomputer implementable method according to claim 10 further comprising;(e) modifying the at least one saliency map in dependence upon at leastthe sequence of pixels; and (f) repeating step (c), (d) and (e) until apredetermined image condition is met, the predetermined image conditionrelating to a target final dimension of the source audiovisual content.14. A computer implementable method according to claim 10 wherein, step(d) comprises the steps: (iv) scaling the sequence of pixels by apredetermined factor; (v) interpolating a path of pixels including thescaled sequence of pixels; and (vi) at least one of removing theinterpolated path of pixels from the source audiovisual content andinserting a sequence of proper pixels into the source audiovisual image,the sequence of proper pixels comprising the interpolated path of pixelswherein each pixel in the interpolated path is replaced by a properpixel determined in dependence upon a process and at least one of the atleast one saliency map and the source audiovisual content.
 15. Acomputer implementable method according to claim 10 wherein, modifyingthe source audiovisual content comprises at least one of inserting andremoving a predetermined number of pixels along a path within the sourceaudiovisual content, the path determined in dependence upon at least thesequence of pixels.
 16. A computer implementable method according toclaim 10 wherein, the at least one predetermined scaling factor isdetermined in dependence upon at least a characteristic of at least oneof the device upon which the computer implementable method is inexecution and the source audiovisual content.
 17. A computerimplementable method according to claim 16 wherein, when the at leastone is the device then the characteristic is selected from the groupcomprising a measure of processor loading, processor type, processorspeed, accessible memory, status, display dimension, display resolution,graphics processor, and battery life and when the at least one is thesource audiovisual content the characteristic is selected from the groupcomprising a file format, a measure of anticipated duration of the imageon the display, and image dimensions.
 18. A computer implementablemethod according to claim 10 wherein, the saliency for the correspondingpixel within the source audiovisual content is determined from at leastan aspect of a color space representation of the pixel.
 19. A computerimplementable method according to claim 10 wherein, a predeterminedportion of at least one of the first data and the second data are atleast one of excluded from, preferentially selected in, and modifiedbefore path determination process.
 20. A device comprising: (a) aninterface for receiving audiovisual content for presentation to a userupon a display forming a predetermined portion of the device, theaudiovisual content characterized by at least a source dimension beingat least one of a physical dimension and a number of pixels; and (b) Acircuit including at least a processor and a memory for executing aseries of processes, the processes including at least: a display processfor determining a target dimension for the audiovisual content forpresentation to the user; and (ii) an image process for generating amodified image in dependence upon at least the audiovisual content, thetarget dimension, and the at least a source dimension, the image processcomprising the steps of: 1) generating a saliency map characterized byat least a number of pixels equal to a number of pixels within thesource audiovisual content and each pixel within the saliency map havingat least two values associated with it, one value determined independence upon at least a saliency for the corresponding pixel withinthe source audiovisual content along a first axis of the image and theother value determined in dependence upon at least a saliency for thecorresponding pixel within the source audiovisual content along a secondaxis of the image; 2) generating a reduced saliency map by applying atleast one predetermined scaling factor to the saliency map, each pixelwith the reduced saliency map having at least first data generated independence upon at least the one value of a pixel within the saliencymap associated with the pixel in the saliency reduced map and seconddata generated in dependence upon at lest the other value of a pixelwithin the saliency map associated with the pixel in the saliencyreduced map; 3) applying a path determination process to at least one ofthe first data and the second data within the reduced saliency map, thepath determination process for identifying a sequence of pixels withinthe reduced saliency map that meet a predetermined condition; and 4)modifying the audiovisual content in dependence upon at least thesequence of pixels to generate display audiovisual content.
 21. A deviceaccording to claim 20 further comprising; (5) modifying the at least onesaliency map in dependence upon at least the sequence of pixels; and (6)repeating steps (c), (d) and (e) until the target dimension has beenachieved.
 22. A device according to claim 20 wherein, step (4) comprisesthe steps: scaling the sequence of pixels by a predetermined factor;interpolating a path of pixels including the scaled sequence of pixels;and at least one of removing the interpolated path of pixels from theaudiovisual content and inserting a sequence of proper pixels into theaudiovisual image, the sequence of proper pixels comprising theinterpolated path of pixels wherein each pixel in the interpolated pathis replaced by a proper pixel determined in dependence upon a processand at least one of the at least one saliency map and the sourceaudiovisual content.
 23. A device according to claim 20 wherein,modifying the audiovisual content comprises at least one of insertingand removing a predetermined number of pixels along a path within theaudiovisual content, the path determined in dependence upon at least thesequence of pixels.
 24. A device according to claim 20 wherein, the atleast one predetermined scaling factor is determined in dependence uponat least a characteristic of at least one of the device and theaudiovisual content.
 25. A device according to claim 24 wherein, whenthe at least one is the device then the characteristic is selected fromthe group comprising a measure of processor loading, processor type,processor speed, accessible memory, status, display dimension, displayresolution, graphics processor, and battery life and when the at leastone is the source audiovisual content the characteristic is selectedfrom the group comprising file format, a measure of anticipated durationof the image on the display, and image dimensions.
 26. A computerimplementable method according to claim 20 wherein, a predeterminedportion of at least one of the first data and the second data are atleast one of excluded from, preferentially selected in, and modifiedbefore path determination process.